

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>The Migration Process &mdash; RWC Converter v1.0 documentation</title>
    <link rel="stylesheet" href="_static/default.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '',
        VERSION:     '1.0',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="top" title="RWC Converter v1.0 documentation" href="index.html" />
    <link rel="next" title="Managing Data Acquisition (with Excel)" href="excel_caution.html" />
    <link rel="prev" title="Preparing for Migration" href="preparation.html" /> 
  </head>
  <body>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="excel_caution.html" title="Managing Data Acquisition (with Excel)"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="preparation.html" title="Preparing for Migration"
             accesskey="P">previous</a> |</li>
        <li><a href="index.html">RWC Converter v1.0 documentation</a> &raquo;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body">
            
  <div class="section" id="the-migration-process">
<span id="id1"></span><h1>The Migration Process<a class="headerlink" href="#the-migration-process" title="Permalink to this headline">¶</a></h1>
<p>There are a few discrete steps involved in the migration of data away from the
Microsoft Word documents.  The ability is provided to write the data into 3 distinct
formats (a delimited text file, XML files, and restructuredText).  The data can also
optionallyl be directly inserted into a relational database such as MySQL. The migration steps are:</p>
<blockquote>
<div><ol class="arabic simple">
<li><a class="reference internal" href="#config-constants"><em>Specify settings</em></a> for the migration.</li>
<li><a class="reference internal" href="#doc-to-docx"><em>Convert files</em></a> from doc to docx.</li>
<li><a class="reference internal" href="#stripping-data"><em>Extract data</em></a> from docx files (called &#8216;stripping&#8217;).</li>
<li><a class="reference internal" href="#other-formats"><em>Write data</em></a> to desired format.</li>
</ol>
</div></blockquote>
<p>All three steps are executed by running the &#8216;main.py&#8217; script in the project root directory.
The script must be run using the python interpreter through the command prompt
(see <a class="reference internal" href="preparation.html#get-started"><em>Preparing for Migration</em></a>). The script uses settings from the &#8216;constants.py&#8217;
file to determine appropriate directories and file names for the migration. Each time the
script is executed, a single command line argument must be provided. Arguments for the script include:</p>
<blockquote>
<div><ul class="simple">
<li><a class="reference internal" href="#doc-to-docx"><em>convert</em></a></li>
<li><a class="reference internal" href="#stripping-data"><em>strip</em></a></li>
<li><a class="reference internal" href="#delim-file"><em>delim-db</em></a></li>
<li><a class="reference internal" href="#xml-files"><em>xml-db</em></a></li>
<li><a class="reference internal" href="#xml-files"><em>xml-files</em></a></li>
<li><a class="reference internal" href="#db-direct"><em>database-direct</em></a></li>
<li><a class="reference internal" href="#rst-output"><em>rst-index</em></a></li>
</ul>
</div></blockquote>
<p>Assuming the Python interpreter is on your system PATH, running the script can be as simple as shown here:</p>
<div class="highlight-python"><pre>python main.py [command]</pre>
</div>
<p>If the current working directory of the command prompt is not the project folder,
<tt class="docutils literal"><span class="pre">main.py</span></tt> above would need to be replaced with <tt class="docutils literal"><span class="pre">\path\to\project\folder\main.py</span></tt>.</p>
<p><em>* Before running any of the procedures described on this page in the command prompt,
the current working directory should be set to the root directory of the migration utility.</em></p>
<div class="section" id="configuring-constants-py">
<span id="config-constants"></span><h2>Configuring constants.py<a class="headerlink" href="#configuring-constants-py" title="Permalink to this headline">¶</a></h2>
<p>The &#8216;constants.py&#8217; file is located in the project root directory. There are several settings
variables contained within a python class of the form <tt class="docutils literal"><span class="pre">[setting_name]</span> <span class="pre">=</span> <span class="pre">'[value]'</span></tt>.  All paths
should be absolute paths (with one exception).  In general, forward slashes should be used when
entering paths.  If back slashes are used, each backslash must be escaped with a backslash
(i.e. <tt class="docutils literal"><span class="pre">\my\absolute\path</span></tt> should be entered as <tt class="docutils literal"><span class="pre">\\my\\absolute\\path</span></tt>). After the variables
have all been given appropriate values, the migration process can begin. A view of the &#8216;constants.py&#8217;
file and a description of the important variables are shown below.  The settings variables will be
referred to throughout this page - often without explicitly identification as a settings variable.</p>
<blockquote>
<div><p><strong>Sample &#8216;constants.py&#8217; file</strong>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c">#!/usr/local/bin/python2.7</span>
<span class="kn">import</span> <span class="nn">os</span>

<span class="k">class</span> <span class="nc">Constants</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
    <span class="n">pile_root</span> <span class="o">=</span> <span class="s">&#39;</span><span class="se">\\\\</span><span class="s">fsisc1</span><span class="se">\\</span><span class="s">projects</span><span class="se">\\</span><span class="s">LIFEEXT</span><span class="se">\\</span><span class="s">atr component database&#39;</span>
    <span class="n">db_root</span> <span class="o">=</span> <span class="s">&#39;C:/docx-final&#39;</span>
    <span class="n">dst_root</span> <span class="o">=</span> <span class="s">&#39;C:/converted-final&#39;</span>
    <span class="n">wordconv_path</span> <span class="o">=</span> <span class="s">&#39;C:/cygwin/home/CALSRW/atr-indexer/resources/wordconv&#39;</span>

    <span class="n">logs_dir</span> <span class="o">=</span> <span class="s">&#39;C:/rwc-logs-final&#39;</span>
    <span class="n">rst_index_dir</span> <span class="o">=</span> <span class="s">&#39;C:/comp-index&#39;</span>

<span class="c">############### shouldn&#39;t need  modification below ###############</span>
    <span class="n">dbase_name</span> <span class="o">=</span> <span class="s">&#39;dbase.pickle&#39;</span>

    <span class="n">temp_files_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">logs_dir</span><span class="p">,</span> <span class="s">&#39;temp&#39;</span><span class="p">)</span>
    <span class="n">pics_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">logs_dir</span><span class="p">,</span> <span class="s">&#39;pics&#39;</span><span class="p">)</span>
    <span class="n">parsed_files_dir</span> <span class="o">=</span> <span class="n">dst_root</span>

    <span class="n">tagdict_filepath</span> <span class="o">=</span> <span class="s">&#39;keys.xml&#39;</span>
</pre></div>
</div>
<p><strong>Settings variables</strong>:</p>
<blockquote>
<div><table border="1" class="docutils">
<colgroup>
<col width="13%" />
<col width="87%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Name</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>pile_root</td>
<td>path to the root directory containing all original Microsoft Word documents</td>
</tr>
<tr><td>db_root</td>
<td>path to the root directory where all Microsoft Word documents will be placed after conversion to docx format</td>
</tr>
<tr><td>dst_root</td>
<td>path to root directory where xml files are stored (if converting to xml format)</td>
</tr>
<tr><td>wordconv_path</td>
<td>path to the &#8216;wordconv.exe&#8217; utility (should be located at <tt class="docutils literal"><span class="pre">[path-to-project]/resources/wordconv</span></tt>)</td>
</tr>
<tr><td>logs_dir</td>
<td>directory where logs and the serialized (pickled) data is stored after stripping from the Word documents</td>
</tr>
<tr><td>rst_index_dir</td>
<td>path where a sphinx project has been created - used for conversion to restructuredText</td>
</tr>
<tr><td>delim_chars</td>
<td>characters used as delimiter for writing data to a delimited file (e.g. csv/comma or otherwise)</td>
</tr>
<tr><td>dbase_name</td>
<td>file name for the serialized (pickled) data after stripping</td>
</tr>
<tr><td>temp_files_dir</td>
<td>temporary directory used to unzip docx data</td>
</tr>
<tr><td>pics_dir</td>
<td>directory where all pictures extracted from Word documents are stored</td>
</tr>
<tr><td>parsed_files_dir</td>
<td>same usage as <tt class="docutils literal"><span class="pre">dst_root</span></tt> (usu. same value as <tt class="docutils literal"><span class="pre">dst_root</span></tt>)</td>
</tr>
<tr><td>tagdict_filepath</td>
<td>(<strong>should be relative path</strong>) path to xml file that contains mappings from Word doc field names to data extraction field names.</td>
</tr>
</tbody>
</table>
</div></blockquote>
</div></blockquote>
</div>
<div class="section" id="doc-to-docx-conversion">
<span id="doc-to-docx"></span><h2>Doc to Docx Conversion<a class="headerlink" href="#doc-to-docx-conversion" title="Permalink to this headline">¶</a></h2>
<p>The actual doc to docx conversion is completed by a Microsoft utility called wordconv.
The migration script is simply a wrapper around the wordconv utility that recursively crawls
through a directory. The conversion handles both &#8216;*.doc&#8217; and &#8216;*.docx&#8217; files appropriately.
&#8216;*.doc&#8217; files are converted to &#8216;*.docx&#8217; and then placed in the specified <tt class="docutils literal"><span class="pre">db_root</span></tt> directory.
&#8216;*.docx&#8217; files are simply copied directly into the specified <tt class="docutils literal"><span class="pre">db_root</span></tt> directory.  To execute the
conversion process, run the &#8216;main.py&#8217; script from the command prompt (cmd.exe) using the
command-line argument <tt class="docutils literal"><span class="pre">convert</span></tt>:</p>
<div class="highlight-python"><pre>python main.py convert</pre>
</div>
<p>To safely terminate the conversion process, press ctrl-c.  If the conversion process terminates
prematurely, restarting it will not reconvert the already converted files. Files that already
exist in the <tt class="docutils literal"><span class="pre">db_root</span></tt> directory will be skipped over and conversion will continue where it left off.
The crawling process does not look at date-modified or date-created meta-data.  If files in the source
(<tt class="docutils literal"><span class="pre">pile_root</span></tt>) directory are modified between conversion runs, the modified file(s) will not be reconverted
and updated in the destination (<tt class="docutils literal"><span class="pre">db_root</span></tt>).</p>
</div>
<div class="section" id="stripping-the-data">
<span id="stripping-data"></span><h2>Stripping the Data<a class="headerlink" href="#stripping-the-data" title="Permalink to this headline">¶</a></h2>
<p>After conversion to docx files, the data can be stripped out and stored in a structured format.
Stripping extracts data and pictures from each Word document.  All pictures are given descriptive, unique
names and placed in a single directory (pics_dir as described above). The data from a single word document
is stored in a Python object (instance of the Component class).  Each object stores data from the word
document in dictionaries (fieldname-value pairs). The object also stores the names of the pictures that
are associated with it.  The collection of all objects representing every word document&#8217;s data
is stored in another umbrella oject (instance of the DBase class).  This object is serialized using python&#8217;s
pickle module and stored as a file named with the value specified by dbase_name (described above).
Custom python scripts can be written relatively easily to manipulate and use this serialized data as desired.</p>
<p>To execute the conversion process, run the &#8216;main.py&#8217; script from the command prompt (cmd.exe) using the
command-line argument <tt class="docutils literal"><span class="pre">strip</span></tt>:</p>
<div class="highlight-python"><pre>python main.py strip</pre>
</div>
<p>To (semi)safely terminate the stripping process, press ctrl-c.  This allows the already processed data and progress
to be saved (to prevent repeating from the beginning). If the conversion process terminates
prematurely, restarting it should not reprocess the already stripped/stored files. However, it is best to complete the
entire stripping process without interruption to ensure a clean stripping (I am only 93% confident in an error
free continuation of a previously interrupted stripping).</p>
</div>
<div class="section" id="outputting-data-to-other-formats">
<span id="other-formats"></span><h2>Outputting Data to Other Formats<a class="headerlink" href="#outputting-data-to-other-formats" title="Permalink to this headline">¶</a></h2>
<p>The serialized data from the stripping process can then be retrieved at will and written to various formats.
Functionality is provided allowing the writing of the data to a few basic formats.  Instructions for writing to
these provided-by-default formats is described in the sections below. Writing the data to other formats has no
effect on the serialized data - it remains intact as originally stripped. Custom python scripts can also be written
relatively easily to manipulate and use the python-serialized (pickled) data as desired (see the &#8216;component.py&#8217; and &#8216;dbase.py&#8217;
source code for this object structure).</p>
<div class="section" id="delimited-file">
<span id="delim-file"></span><h3>Delimited File<a class="headerlink" href="#delimited-file" title="Permalink to this headline">¶</a></h3>
<p>The data can be written to a comma separated values file (csv) or any other delimited file.  The delimiter can be a series of characters of any length.
The default delimiter is &#8216;$&#8217; (chosen because it is not present in any of the word documents). Alternate delimiters can be used by specifying them in the
delim_chars variable of the constants.py file. To write the data to delimited file, run the &#8216;main.py&#8217; script from the command prompt (cmd.exe) using the
command-line argument <tt class="docutils literal"><span class="pre">delim-db</span></tt>:</p>
<div class="highlight-python"><pre>python main.py delim-db</pre>
</div>
<p>The resulting delimited file is written to the directory specified by the <tt class="docutils literal"><span class="pre">logs_dir</span></tt> variable setting
The file is named &#8216;mycmdb.txt&#8217; by default. These defaults can be altered if desired in the &#8216;delimwriter.py&#8217; file.
The resulting delimited file can then be imported into Excel (Excel only supports importing single character
delimited files) or used as desired.</p>
</div>
<div class="section" id="xml-files">
<span id="id2"></span><h3>XML Files<a class="headerlink" href="#xml-files" title="Permalink to this headline">¶</a></h3>
<p>The data can be written to a set of xml files where each component&#8217;s data is stored in a uniquely named xml file.
The collection of xml files are placed into a single directory as specified by the <tt class="docutils literal"><span class="pre">dst_root</span></tt> setting.
To write the data to xml files, run the &#8216;main.py&#8217; script from the command prompt (cmd.exe) using the
command-line argument <tt class="docutils literal"><span class="pre">xml-files</span></tt>:</p>
<div class="highlight-python"><pre>python main.py xml-files</pre>
</div>
<p>The data can also be written to a single xml file that is simply the consolidation of the individual xml files as described above.
The resulting xml file is written to the directory specified by the <tt class="docutils literal"><span class="pre">logs_dir</span></tt> variable setting
The file is named &#8216;mycmdb.xml&#8217; by default. These defaults can be altered if desired in the &#8216;xmldb.py&#8217; file.
The resulting xml file can then be imported into a database, viewed in a browser, or used otherwise.
To write the data to a single, consolidated xml file, run the &#8216;main.py&#8217; script from the command prompt (cmd.exe) using the
command-line argument <tt class="docutils literal"><span class="pre">xml-db</span></tt>:</p>
<div class="highlight-python"><pre>python main.py xml-db</pre>
</div>
</div>
<div class="section" id="direct-database-insertion">
<span id="db-direct"></span><h3>Direct Database Insertion<a class="headerlink" href="#direct-database-insertion" title="Permalink to this headline">¶</a></h3>
<p><em>This functionality requires the django framework to be added to your python installation as
described in</em> <a class="reference internal" href="preparation.html#install-django"><em>Adding Django</em></a>.</p>
<p>There must be an accessible, supported database server running for this to work.
Supported databases include PostgreSQL, MySQL, SQLite, and Oracle.
The Django framework is used as an abstraction layer for the database.
A set of tables are created in the database with appropriate relationships.
The data for each component is inserted into these tables.</p>
<p>The django files in the migration project reside in <tt class="docutils literal"><span class="pre">[path</span> <span class="pre">to</span> <span class="pre">project]/djangodb</span></tt> directory.
In order to connect to a database server, the &#8216;settings.py&#8217; file in the &#8216;djangodb&#8217; directory
must be configured properly. The section of the file that needs to be edited is shown below.
The ENGINE, NAME, USER, PASSWORD, HOST, and PORT for the database server must be specified.</p>
<p>Django&#8217;s settings.py file:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="o">...</span>
<span class="o">...</span>
<span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s">&#39;default&#39;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s">&#39;ENGINE&#39;</span><span class="p">:</span> <span class="s">&#39;django.db.backends.mysql&#39;</span><span class="p">,</span> <span class="c"># Add &#39;postgresql_psycopg2&#39;, &#39;postgresql&#39;, &#39;mysql&#39;, &#39;sqlite3&#39; or &#39;oracle&#39;.</span>
        <span class="s">&#39;NAME&#39;</span><span class="p">:</span> <span class="s">&#39;django-fun&#39;</span><span class="p">,</span>                 <span class="c"># Or path to database file if using sqlite3.</span>
        <span class="s">&#39;USER&#39;</span><span class="p">:</span> <span class="s">&#39;root&#39;</span><span class="p">,</span>                       <span class="c"># Not used with sqlite3.</span>
        <span class="s">&#39;PASSWORD&#39;</span><span class="p">:</span> <span class="s">&#39;password&#39;</span><span class="p">,</span>               <span class="c"># Not used with sqlite3.</span>
        <span class="s">&#39;HOST&#39;</span><span class="p">:</span> <span class="s">&#39;&#39;</span><span class="p">,</span>                           <span class="c"># Set to empty string for localhost. Not used with sqlite3.</span>
        <span class="s">&#39;PORT&#39;</span><span class="p">:</span> <span class="s">&#39;&#39;</span><span class="p">,</span>                           <span class="c"># Set to empty string for default. Not used with sqlite3.</span>
    <span class="p">}</span>
<span class="p">}</span>
<span class="o">...</span>
<span class="o">...</span>
</pre></div>
</div>
<p>The values shown above would be correct if connecting to a MySQL server with a database named
&#8216;django-fun&#8217; hosted on it.  The host and port can be left blank if the database is using defaults for them.
The direct insertion can be executed by running the &#8216;main.py&#8217; script  from the command prompt using the
command-line argument <tt class="docutils literal"><span class="pre">database-direct</span></tt>:</p>
<div class="highlight-python"><pre>python main.py database-direct</pre>
</div>
</div>
<div class="section" id="restructuredtext-writer">
<span id="rst-output"></span><h3>restructuredText Writer<a class="headerlink" href="#restructuredtext-writer" title="Permalink to this headline">¶</a></h3>
<p><em>The Sphinx framework must be added to your python installation as described</em> <a class="reference internal" href="preparation.html#install-sphinx"><em>here</em></a>.</p>
<p>This output format was created for my convenience during development.  If you can figure out how it works, good for you.</p>
</div>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar">
        <div class="sphinxsidebarwrapper">
  <h3><a href="index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">The Migration Process</a><ul>
<li><a class="reference internal" href="#configuring-constants-py">Configuring constants.py</a></li>
<li><a class="reference internal" href="#doc-to-docx-conversion">Doc to Docx Conversion</a></li>
<li><a class="reference internal" href="#stripping-the-data">Stripping the Data</a></li>
<li><a class="reference internal" href="#outputting-data-to-other-formats">Outputting Data to Other Formats</a><ul>
<li><a class="reference internal" href="#delimited-file">Delimited File</a></li>
<li><a class="reference internal" href="#xml-files">XML Files</a></li>
<li><a class="reference internal" href="#direct-database-insertion">Direct Database Insertion</a></li>
<li><a class="reference internal" href="#restructuredtext-writer">restructuredText Writer</a></li>
</ul>
</li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="preparation.html"
                        title="previous chapter">Preparing for Migration</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="excel_caution.html"
                        title="next chapter">Managing Data Acquisition (with Excel)</a></p>
  <h3>This Page</h3>
  <ul class="this-page-menu">
    <li><a href="_sources/the_conversion_process.txt"
           rel="nofollow">Show Source</a></li>
  </ul>
<div id="searchbox" style="display: none">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <input type="text" name="q" size="18" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    <p class="searchtip" style="font-size: 90%">
    Enter search terms or a module, class or function name.
    </p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="excel_caution.html" title="Managing Data Acquisition (with Excel)"
             >next</a> |</li>
        <li class="right" >
          <a href="preparation.html" title="Preparing for Migration"
             >previous</a> |</li>
        <li><a href="index.html">RWC Converter v1.0 documentation</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer">
        &copy; Copyright 2011, Robert Carlsen.
      Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.0.7.
    </div>
  </body>
</html>